Predict Default of Credit Card Clients¶
Predict Default of Credit Card Clients by customer last months data
2. Dataset¶
This dataset contains information on default payments, demographic factors, credit data, history of payment, and bill statements of credit card clients in Taiwan from April 2005 to September 2005. There are 25 variables:
ID: ID of each client
LIMIT_BAL: Amount of given credit in NT dollars (includes individual and family/supplementary credit
- SEX: Gender (1=male, 2=female)
- EDUCATION: (1=graduate school, 2=university, 3=high school, 4=others, 5=unknown, 6=unknown)
- MARRIAGE: Marital status (1=married, 2=single, 3=others)
- AGE: Age in years
- PAY_0: Repayment status in September, 2005 (-1=pay duly, 1=payment delay for one month, 2=payment delay for two months, …
- 8=payment delay for eight months, 9=payment delay for nine months and above)
- PAY_2: Repayment status in August, 2005 (scale same as above)
- PAY_3: Repayment status in July, 2005 (scale same as above)
- PAY_4: Repayment status in June, 2005 (scale same as above)
- PAY_5: Repayment status in May, 2005 (scale same as above)
- PAY_6: Repayment status in April, 2005 (scale same as above)
- BILL_AMT1: Amount of bill statement in September, 2005 (NT dollar)
- BILL_AMT2: Amount of bill statement in August, 2005 (NT dollar)
- BILL_AMT3: Amount of bill statement in July, 2005 (NT dollar)
- BILL_AMT4: Amount of bill statement in June, 2005 (NT dollar)
- BILL_AMT5: Amount of bill statement in May, 2005 (NT dollar)
- BILL_AMT6: Amount of bill statement in April, 2005 (NT dollar)
- PAY_AMT1: Amount of previous payment in September, 2005 (NT dollar)
- PAY_AMT2: Amount of previous payment in August, 2005 (NT dollar)
- PAY_AMT3: Amount of previous payment in July, 2005 (NT dollar)
- PAY_AMT4: Amount of previous payment in June, 2005 (NT dollar)
- PAY_AMT5: Amount of previous payment in May, 2005 (NT dollar)
- PAY_AMT6: Amount of previous payment in April, 2005 (NT dollar)
- default.payment.next.month: Default payment (1=yes, 0=no)
In [ ]:
3.1 Data Explore¶
- Load the liberaries
- Load the data
- correct data "data types"
- Check duplication
- Check missing data
- Check NAN/zero data
- Check data statstics
- check unique values for the catogorical columns
- Check the data ditrebution for the catogorical data
- Check data balance
- Check data features correlation
In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.offline as py
py.init_notebook_mode(connected=True)
from plotly.offline import init_notebook_mode, iplot
init_notebook_mode(connected=True)
import plotly.graph_objs as go
import plotly.offline as offline
offline.init_notebook_mode()
import cufflinks as cf
cf.go_offline()
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, make_scorer
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.metrics import roc_auc_score
from sklearn.metrics import precision_recall_fscore_support
In [2]:
import pandas as pd # package for high-performance, easy-to-use data structures and data analysis
import numpy as np # fundamental package for scientific computing with Python
import matplotlib
import matplotlib.pyplot as plt # for plotting
import seaborn as sns # for making plots with seaborn
color = sns.color_palette()
import plotly.offline as py
py.init_notebook_mode(connected=True)
from plotly.offline import init_notebook_mode, iplot
init_notebook_mode(connected=True)
import plotly.graph_objs as go
import plotly.offline as offline
offline.init_notebook_mode()
# from plotly import tools
# import plotly.tools as tls
# import squarify
# from mpl_toolkits.basemap import Basemap
# from numpy import array
# from matplotlib import cm
# import cufflinks and offline mode
import cufflinks as cf
cf.go_offline()
In [3]:
#Read the data from xls file with sheet nema : Data
data=pd.read_excel('default-of-credit-card-clients.xls', 'Data', index_col=0, na_values=['NA'],)
In [4]:
new_header =data.iloc[0]
data = data[1:]
data.columns = new_header
In [5]:
data.head()
Out[5]:
In [6]:
print('Number of rows '+ str(data.shape[0]))
print('Number of cloumns '+ str(data.shape[1]))
In [7]:
data.info()
In [8]:
#correct data "data types"
data = data.astype({"PAY_0":'float',"PAY_2":'float',"PAY_3":'float',"PAY_4":'float',"PAY_5":'float',"PAY_6":'float',"AGE":'float', "BILL_AMT1":'float',"BILL_AMT2":'float',"BILL_AMT3":'float',
"BILL_AMT4":'float',"BILL_AMT5":'float',"BILL_AMT6":'float',"PAY_AMT1":'float',
"PAY_AMT2":'float',"PAY_AMT3":'float',"PAY_AMT4":'float',"PAY_AMT5":'float',"PAY_AMT6":'float'
,"LIMIT_BAL":'float',"default payment next month":'float'})
In [9]:
data.info()
In [10]:
#check data statstics
data.describe()
Out[10]:
In [11]:
model_features = data.columns.drop('default payment next month')
model_target = 'default payment next month'
In [12]:
data.nunique()
Out[12]:
In [13]:
data.isna().sum()
Out[13]:
In [14]:
#check unique values for the catogorical columns
for column in data.select_dtypes(include=['object']).columns:
print(data[column].unique())
In [15]:
#check the duplicaation in the data
duplicates = data[data.duplicated()]
len(duplicates)
Out[15]:
In [16]:
#check data balance
data[model_target].value_counts()/data.shape[0]
Out[16]:
In [17]:
#check the data ditrebution for the catogorical data
for column in data.select_dtypes(include=['object']).columns:
display(pd.crosstab(index=data[column], columns='% observations', normalize='columns')*100)
3.1.1 Data Explore - findings¶
- the data has 30000 observations and 24 features
- all data features are object type . so, i have to convert the columns that have numeric values to numeric type to be able to get more insights from the data
- The catogrical data has NANs and zeros value must be imputed or deleted
- Data has 35 duplicates rows must be deleted
- "PAY_" cloums have values not defind in data description which are(0,-2). as per kaggle discussions -2 -> "No consumption" and 0 -> "The use of revolving credit"
- there is an outlier in Age Feture which is some of records have age more than 100 year
- There ara record in "BILL_" columns with negative values "which is weired". as per kaggle discussions It is generally possible for a credit card customer to overpay their bill and temporarily carry a negative balance
- The data has 77% with default payment flag with value 0 and 22% with default payment flag with value 1. which means the data labels imbalanced
In [18]:
#data balance from target prespective
temp = data["default payment next month"].value_counts()
df = pd.DataFrame({'labels': temp.index,
'values': temp.values
})
df.iplot(kind='pie',labels='labels',values='values', title='default payment or not')
In [19]:
temp = data["AGE"]
temp.iplot(kind='histogram', title='Age Distribution')
In [20]:
temp = data["EDUCATION"].value_counts()
df = pd.DataFrame({'labels': temp.index,
'values': temp.values
})
df.iplot(kind='pie',labels='labels',values='values', title='EDUCATION Status ', hole = 0.5)
In [21]:
temp = data["MARRIAGE"].value_counts()
df = pd.DataFrame({'labels': temp.index,
'values': temp.values
})
df.iplot(kind='pie',labels='labels',values='values', title='MARRIAGE Status ', hole = 0.5)
In [22]:
plt.figure(figsize = (40,20))
sns.heatmap(data.corr(),annot = True,square = True)
Out[22]:
In [ ]:
In [23]:
#Remove Duplicated data
clean_data=data.drop_duplicates()
In [24]:
#drop NAN data
clean_data=clean_data.dropna()
In [25]:
#remove the outliers
clean_data=clean_data[clean_data['AGE'] <= 100]
In [26]:
temp = clean_data["AGE"]
temp.iplot(kind='histogram', title='Age Distribution')
In [27]:
#remove the unassigend EDUCATION
clean_data=clean_data[clean_data['EDUCATION'] != 0]
In [28]:
#remove the unassigned MARRIAGE status
clean_data=clean_data[clean_data['MARRIAGE'] != 0]
In [29]:
for column in clean_data.select_dtypes(include=['object']).columns:
print(clean_data[column].unique())
In [30]:
#convert catogerical data to numerical by applying on-hot encoding
categorical_columns =['SEX','EDUCATION','MARRIAGE']
data_dummies = pd.get_dummies(clean_data[categorical_columns], drop_first=True)
clean_data = pd.concat([clean_data, data_dummies], axis = 1)
clean_data.drop(categorical_columns,axis=1, inplace=True)
In [31]:
# crearte another data but with appling minmax scaler
from sklearn import preprocessing
minmax= preprocessing.MinMaxScaler().fit(clean_data)
minmax_clean_data=minmax.transform(clean_data)
minmax_clean_data=pd.DataFrame(minmax_clean_data,columns=list(clean_data))
In [32]:
# crearte another data but with upsampling the data to solve imbalanced data
from sklearn.utils import resample
#minmax_Y = minmax_clean_data['default payment next month'].copy()
#minmax_X = minmax_clean_data[features].copy()
maj_data=minmax_clean_data[minmax_clean_data['default payment next month']==0]
min_data=minmax_clean_data[minmax_clean_data['default payment next month']==1]
min_data_oversample=resample(min_data,replace=True,n_samples=22000,random_state=587)
oversample_minmax_clean_data=pd.concat([maj_data,min_data_oversample])
oversample_minmax_clean_data['default payment next month'].value_counts()
Out[32]:
In [33]:
#data balance from target prespective
temp = oversample_minmax_clean_data["default payment next month"].value_counts()
df = pd.DataFrame({'labels': temp.index,
'values': temp.values
})
df.iplot(kind='pie',labels='labels',values='values', title='default payment or not')
In [34]:
features = ['LIMIT_BAL', 'SEX_male', 'EDUCATION_high school','EDUCATION_university','EDUCATION_others', 'MARRIAGE_single', 'AGE', 'PAY_0', 'PAY_2',
'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2',
'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1',
'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']
In [35]:
#Train/Test data for cleaned dataset
clean_Y = clean_data['default payment next month'].copy()
clean_X = clean_data[features].copy()
clean_X_train, clean_X_test, clean_y_train, clean_y_test = train_test_split(clean_X,clean_Y, test_size=0.20,shuffle=True, random_state=42)
In [36]:
#Train/Test data for cleaned and minmax scaled dataset
minmax_clean_Y = minmax_clean_data['default payment next month'].copy()
minmax_clean_X = minmax_clean_data[features].copy()
minmax_clean_X_train, minmax_clean_X_test, minmax_clean_y_train, minmax_clean_y_test = train_test_split(minmax_clean_X,minmax_clean_Y, test_size=0.20,shuffle=True, random_state=42)
In [37]:
#Train/Test data for cleaned and minmax scaled nand upsampled dataset
oversample_minmax_clean_Y = oversample_minmax_clean_data['default payment next month'].copy()
oversample_minmax_clean_X = oversample_minmax_clean_data[features].copy()
oversample_minmax_clean_X_train, oversample_minmax_clean_X_test, oversample_minmax_clean_y_train, oversample_minmax_clean_y_test = train_test_split(oversample_minmax_clean_X,oversample_minmax_clean_Y, test_size=0.20,shuffle=True, random_state=42)
In [38]:
#DecisionTreeClassifier with tree depth =2 and cleaned data
classifier = DecisionTreeClassifier(max_depth=2, random_state=14)
# fit the classifier
classifier.fit(clean_X_train, clean_y_train)
# test the predictions
predictions = classifier.predict(clean_X_test)
In [39]:
cf=confusion_matrix(clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
In [40]:
#DecisionTreeClassifier with tree depth =2 and cleaned-minmax scale data
classifier = DecisionTreeClassifier(max_depth=2, random_state=14)
# fit the classifier
classifier.fit(minmax_clean_X_train, minmax_clean_y_train)
# test the predictions
predictions = classifier.predict(minmax_clean_X_test)
In [41]:
cf=confusion_matrix(minmax_clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = minmax_clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(minmax_clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(minmax_clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
In [42]:
#DecisionTreeClassifier with tree depth =2 and cleaned-minmax scale and updampled data
classifier = DecisionTreeClassifier(max_depth=2, random_state=14)
# fit the classifier
classifier.fit(oversample_minmax_clean_X_train, oversample_minmax_clean_y_train)
# test the predictions
predictions = classifier.predict(oversample_minmax_clean_X_test)
accuracy_score(y_true = oversample_minmax_clean_y_test, y_pred = predictions)
Out[42]:
In [43]:
cf=confusion_matrix(oversample_minmax_clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = oversample_minmax_clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(oversample_minmax_clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(oversample_minmax_clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
In [44]:
#DecisionTreeClassifier with tree depth =32 and cleaned-minmax scale and updampled data
classifier = DecisionTreeClassifier(max_depth=32, random_state=14)
# fit the classifier
classifier.fit(oversample_minmax_clean_X_train, oversample_minmax_clean_y_train)
# test the predictions
predictions = classifier.predict(oversample_minmax_clean_X_test)
In [45]:
cf=confusion_matrix(oversample_minmax_clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = oversample_minmax_clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(oversample_minmax_clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(oversample_minmax_clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
5.1.1 Decision Tree -findings ¶
- applying mimax scaled data not afffect the model accurace . it is the same as orignal data.
- for original / scaled data more tree depth more overfititng for the model
- upsampled data do better with more tree depth
- Maximum Accuracy 0.87
In [46]:
from sklearn import linear_model
In [47]:
#LogisticRegression modle for cleaned data
logreg =linear_model.LogisticRegression()
logreg.fit(clean_X_train, clean_y_train)
# test the predictions
predictions = logreg.predict(minmax_clean_X_test)
accuracy_score(y_true = clean_y_test, y_pred = predictions)
Out[47]:
In [48]:
cf=confusion_matrix(clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
In [49]:
#LogisticRegression modle for cleaned data
logreg =linear_model.LogisticRegression()
# fit the classifier
logreg.fit(minmax_clean_X_train, minmax_clean_y_train)
# test the predictions
predictions = logreg.predict(minmax_clean_X_test)
In [50]:
cf=confusion_matrix(minmax_clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = minmax_clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(minmax_clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
In [51]:
#LogisticRegression modle for cleaned data
logreg =linear_model.LogisticRegression()
# fit the classifier
logreg.fit(oversample_minmax_clean_X_train, oversample_minmax_clean_y_train)
# test the predictions
predictions = logreg.predict(oversample_minmax_clean_X_test)
In [52]:
cf=confusion_matrix(oversample_minmax_clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = oversample_minmax_clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(oversample_minmax_clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(oversample_minmax_clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
5.2.1 Logistic Regression - Findings¶
- mimax scaled data same model accurace as orignal data.
- upsampled data did not do well with logistic regression model. my be i need to adjust few hyper parameters!!
- Maximum accuracy 0.8
In [53]:
from sklearn.neighbors import KNeighborsClassifier
In [54]:
#LogisticRegression modle for cleaned data
knn=KNeighborsClassifier(n_neighbors=10)
knn.fit(clean_X_train, clean_y_train)
# test the predictions
predictions = knn.predict(clean_X_test)
In [55]:
cf=confusion_matrix(clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
In [56]:
#LogisticRegression modle for cleaned data
knn=KNeighborsClassifier(n_neighbors=10)
knn.fit(minmax_clean_X_train, minmax_clean_y_train)
# test the predictions
predictions = knn.predict(minmax_clean_X_test)
In [57]:
cf=confusion_matrix(minmax_clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = minmax_clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(minmax_clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(minmax_clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
In [58]:
#LogisticRegression modle for cleaned data
knn=KNeighborsClassifier(n_neighbors=1)
knn.fit(oversample_minmax_clean_X_train, oversample_minmax_clean_y_train)
# test the predictions
predictions = knn.predict(oversample_minmax_clean_X_test)
In [59]:
cf=confusion_matrix(oversample_minmax_clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = oversample_minmax_clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(oversample_minmax_clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(oversample_minmax_clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
5.3.1 KNN - Findings¶
- MinMax scaled data work better with KNN than original data
- for original data and MinMax scaled data , more neigbors number more effecient prediction
- but for upsampled data , less neigbors numbers more effecient prediction
- Maximum Accuracy 0.87
In [60]:
from sklearn.ensemble import RandomForestClassifier
In [61]:
rf=RandomForestClassifier(n_jobs=10,random_state=10, n_estimators=20, verbose=False)
rf.fit(clean_X_train, clean_y_train)
#test the predictions
predictions = rf.predict(clean_X_test)
In [62]:
cf=confusion_matrix(clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
In [63]:
rf=RandomForestClassifier(n_jobs=10,random_state=10, n_estimators=20, verbose=False)
rf.fit(minmax_clean_X_train, minmax_clean_y_train)
#test the predictions
predictions = rf.predict(minmax_clean_X_test)
In [64]:
cf=confusion_matrix(minmax_clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = minmax_clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(minmax_clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(minmax_clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
In [65]:
rf=RandomForestClassifier(n_jobs=10,random_state=20, n_estimators=20, verbose=False)
rf.fit(oversample_minmax_clean_X_train, oversample_minmax_clean_y_train)
#test the predictions
predictions = rf.predict(oversample_minmax_clean_X_test)
In [66]:
cf=confusion_matrix(oversample_minmax_clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = oversample_minmax_clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(oversample_minmax_clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(oversample_minmax_clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
5.4.1 Random Forrest - Findings¶
- mimax scaled data same model accurace as orignal data.
- upsampled data do way better with random forest model than other data
- Maximum accuracy 0.93
In [67]:
from xgboost import XGBClassifier
In [68]:
# fit model no training data
xgb = XGBClassifier(use_label_encoder=False,
learning_rate =0.1,
n_estimators=1000,
max_depth=100,
min_child_weight=1,
gamma=0,
subsample=0.8,
colsample_bytree=0.8,
objective= 'binary:logistic',
nthread=4,
scale_pos_weight=1,
seed=27)
xgb.fit(clean_X_train, clean_y_train)
#test the predictions
predictions = xgb.predict(clean_X_test)
In [69]:
cf=confusion_matrix(clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
In [70]:
# fit model no training data
xgb = XGBClassifier(use_label_encoder=False,
learning_rate =0.1,
n_estimators=1000,
max_depth=100,
min_child_weight=1,
gamma=0,
subsample=0.8,
colsample_bytree=0.8,
objective= 'binary:logistic',
nthread=4,
scale_pos_weight=1,
seed=27)
xgb.fit(minmax_clean_X_train, minmax_clean_y_train)
#test the predictions
predictions = xgb.predict(minmax_clean_X_test)
In [71]:
cf=confusion_matrix(minmax_clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = minmax_clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(minmax_clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(minmax_clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
In [72]:
# fit model no training data
xgb = XGBClassifier(use_label_encoder=False,
learning_rate =0.1,
n_estimators=1000,
max_depth=100,
min_child_weight=1,
gamma=0,
subsample=0.8,
colsample_bytree=0.8,
objective= 'binary:logistic',
nthread=4,
scale_pos_weight=1,
seed=27)
xgb.fit(oversample_minmax_clean_X_train, oversample_minmax_clean_y_train)
#test the predictions
predictions = xgb.predict(oversample_minmax_clean_X_test)
In [73]:
cf=confusion_matrix(oversample_minmax_clean_y_test, predictions)
sns.heatmap(cf, annot=True, fmt='.0f', cmap="YlGnBu").set_title('Confusion Matrix')
accuracy = accuracy_score(y_true = oversample_minmax_clean_y_test, y_pred = predictions)
print(f'Accuracy: {accuracy}')
auc = roc_auc_score(oversample_minmax_clean_y_test, predictions)
print(f'AUC : {auc}')
precision, recall, f1_score, _ = precision_recall_fscore_support(oversample_minmax_clean_y_test, predictions, average = 'binary')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1_score: {f1_score}')
5.5.1 XGBoost - Findings¶
- original data and minmax data has same accuracy with xdboost
- Upsampled data worked very good with xgboost
- more max depth more accuracy
- maximum accuracy 0.93, we c an get more acuuracy but with more hyperparameter tuning
------------------------------------------------------------------------------------------------¶
6. Conclusion¶
- Feature Engineering helped me alot to improve model accuracy
- balanced data way better than imbalanced data in model train/test
- Accuracy from best to worst (Random Forest > XGBoost > KNN > Decision Tree > Logistic Regression)
7. Improvement Ideas¶
- Use SMOTE oversampling technique rather than sklearn resample
- use hyperopt to tune models hyperparameter